Summary

Production TPD has been restarting every 30-40 seconds (RestartCount: 556 over 6 hours on the prod host running develop tip 96ee3806c). Two distinct panics, found by greping the container logs, are tearing down the process:

1. pkg/cxo/skyobject/cache.go — (*Cache).Finc to negative

Stack:

panic: Finc to negative for: <hash>
github.com/skycoin/skywire/pkg/cxo/skyobject.(*Cache).Finc cache.go:1189
github.com/skycoin/skywire/pkg/cxo/skyobject.(*Filler).apply fill.go:225
github.com/skycoin/skywire/pkg/cxo/skyobject.(*Filler).Run.func1 fill.go:297

The filling-refcount goes below zero — likely a duplicate Finc on a Filler.incs map, or an Inc/Finc mismatch across overlapping fillers. The historical behavior was to panic, which became visible after #2420 increased the rate of fill churn (or the bug was always latent and load surfaced it).

Filler.apply and Filler.reject already consume Finc's error return and just log:

if err := f.c.Finc(key, inc); err != nil {
    log.Printf("[CXO] filler apply: failed to increment ref count for %s: %v", key.Hex(), err)
}

So surfacing the inconsistency through that path costs nothing extra. The fix clamps fc to 0, logs the offending key + inc, and continues. Worst case is a leaked filling-item slot — orders of magnitude better than a process kill every 30 seconds.

2. pkg/httputil/httputil.go — WriteJSON panic on slow-client timeout

Stack:

panic: short write: i/o deadline reached
github.com/skycoin/skywire/pkg/httputil.WriteJSON httputil.go:50
github.com/skycoin/skywire/pkg/transport-discovery/api.(*API).getAllTransports endpoints.go:319

getAllTransports writes ~1MB JSON. On a slow client, net/http's per-write deadline expires mid-response, and the response writer returns an error whose message is "short write: i/o deadline reached". The body is io.ErrShortWrite's text, but the value is net/http.timeoutWriter's own error — errors.Is(err, io.ErrShortWrite) returns false. The fallback string match in isIOError only checked "broken pipe" and "connection reset", so the discriminator returned false and WriteJSON ran the panic branch.

Added "short write", "i/o timeout", and "deadline exceeded" to the string-match fallback. TestIsIOErrorShortWriteVariants pins both sentinel and string-match cases.

Test plan

• go build ./... clean.
• go vet ./pkg/cxo/... ./pkg/httputil/... clean.
• go test ./pkg/httputil/ ./pkg/cxo/skyobject/ ./pkg/transport-discovery/... all pass.
• gofmt clean.
• New unit test TestIsIOErrorShortWriteVariants covers nil, all four current sentinels, the production short-write case, the i/o-timeout / deadline-exceeded text variants, and the existing broken-pipe / connection-reset cases.
• Post-deploy: TPD container RestartCount no longer climbing; docker logs transport-discovery | grep -c panic: reaches 0 within a few minutes.

Notes

Neither bug is caused by the recent latency PRs (#2415, #2418, #2421); the panics' stacks make that explicit. They've just been hammering the deployment more visibly since latency persistence and outlier filtering landed back-to-back, because every TPD bounce now drops registration TTLs on visors that haven't re-pushed yet. Fixing these two restores normal TPD uptime.